Sirtiun 5 (SIRT5) is a NAD+-dependent protein lysine deacylase primarily located in mitochondria. SIRT5 displays an affinity for negatively charged acyl groups and mainly catalyzes lysine deglutarylation, desuccinylation, and demalonylation while possessing weak deacetylase activity. SIRT5 substrates play crucial roles in metabolism and reactive oxygen species (ROS) detoxification, and SIRT5 activity is protective in neuronal and cardiac physiology. Moreover, SIRT5 exhibits a dichotomous role in cancer, acting as context-dependent tumor promoter or suppressor. Given its multifaceted activity, SIRT5 is a promising target in the design of activators or inhibitors that might act as therapeutics in many pathologies, including cancer, cardiovascular disorders, and neurodegeneration. To date, few cellular-active peptide-based SIRT5 inhibitors (SIRT5i) have been described, and potent and selective small-molecule SIRT5i have yet to be discovered. In this perspective, we provide an outline of SIRT5's roles in different biological settings and describe SIRT5 modulators in terms of their mode of action, pharmacological activity, and structure-activity relationships.